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Given a general channel b over which the uniform X source, 
denoted by U, is directly communicated within distortion D. The 
source U puts uniform distribution on all sequences with type pre¬ 
cisely px as compared with the i.i.d. X source which puts ‘most 
of’ its mass on sequences with type ‘close to’ px- A randomized 
covering-packing duality is established between source-coding and 
channel-coding by considering the source-coding problem (cover¬ 
ing problem) of coding the source U within distortion D and the 
channel coding problem (packing problem) of reliable communica¬ 
tion over b, thus leading to a proof of C > Ru{D) where C is the 
capacity of b and Ru{D) is the rate-distortion function of U. This 
also leads to an operational view of source-channel separation for 
communication with a fidelity criterion. 
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1. Introduction 


Given a general channel b over which the uniform X source is directly 
communicated within distortion D. 

This means the following: 

Let the source input space be X and the source reproduction space be y. 
X and y are finite sets. Intuitively, a uniform X source, U, puts a uniform 
distribution on all sequences with a type px- This is as opposed to the i.i.d. 

*Footnote to the title with the ‘thankstext’ command. 
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Figure 1: A channel which communicates the uniform X source within dis¬ 
tortion D 


X source which puts “most of” its mass on sequences with type “close to” 
px- See Section 3 for a precise definition. A general channel is a sequence 
< ig a, transition probability from to a precise 

definition of a general channel can be found in Section 3. When the block- 
length is n, the uniform X source is denoted by C/”. With input into the 
channel, the output is and is such that 

(1) lim Pr ( y”) > = 0 

n—>-oo yn J 

where < : X'^ x —)> [0, oo), is a permutation-invariant (a 

special case is additive) distortion function. The generality of the channel 
is in the sense of Verdu and Han [1]. See Section 3 for precise definitions. 
See Figure 1. 

Such a general channel intuitively functions as follows: when the block- 
length is n, with high probability, a sequence in y G is distorted within 
a ball of radius nD and this probability —)• 0 as n —)■ oo. Note that vT" G 
but the ball of radius nD exists in the output space y^. See Figure 2. 

Note that the uniform X source is not defined for all block-lengths; this point 
will be clarified in Section 3. 

Consider the two problems: 

• Covering problem: the rate-distortion source-coding problem of com¬ 
pressing the source U within distortion D, that is, computing the min¬ 
imum rate needed to compress the source U within a distortion D. De¬ 
note the rate-distortion function by Ru{D). Intuitively, the question 
is to find the minimum number of y'^ G such that balls of radii 
nD circled around y” cover the space lA'^. Note that balls are circled 
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Figure 2: Intuitive action of a channel which directly communicates the 
uniform X source within a distortion D 
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on y” G but balls of radius nD exist in Since the setting is 
information-theoretic, the balls should ‘almost’ cover the whole space. 

• Packing problem; the channel-coding problem of communicating reli¬ 
ably over a general channel b which is known to directly communicate 
the source U within distortion D (packing problem). Denote the chan¬ 
nel capacity by C. Intuitively, the question is to find the maximum 
number of tt” G such that balls of radii nD circled around these 
tt” pack the 3^” space. Note that u” G but balls of radii nD cir¬ 
cled around these codewords exist in the 3^™ space. Since the setting 
is information theoretic, the balls which pack the space can overlap ‘a 
bit’. 

Clearly, there is a duality in these problem statements. It is unclear how 
to make this duality precise for these deterministic problems. However, a 
randomized covering-packing duality can be established between the above 
two problems, thus also proving that the answer to the first problem is less 
than or equal to the answer to the second problem, in the following way: 

The codebook construction and error analysis for the source-coding problem 
are roughly the following: Let the block-length be n. Generate 2”^ codewords 
G independently and uniformly from the set of all sequences with type 
q where q is an achievable type on the output space. Roughly, a u” G 
is encoded via minimum distance encoding. The main error analysis which 
needs to be carried out is the probability that a certain codebook sequence 
does not encode a particular n”, that is, 

(2) PT(-d^{u^,Y^)>D 

\n 

where T” is a uniform random variable on sequences of type q. A best 
possible q is chosen in order to get an upper bound on the rate-distortion 
function. 

The codebook construction and error analysis for the channel-coding prob¬ 
lem are roughly the following: Let the block length be n. Generate 2”^ 
codewords G independently and uniformly. Let y” be received. The de¬ 
coding of which codeword is transmitted is roughly via minimum distance 
decoding. As will become clearer later, the main error calculation in the 
channel-coding problem is the probability of correct decoding for which the 
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following needs to be calculated: 

(3) ¥T(-r{U^,y^)>D 

\n 

where y" has type q. Finally, a worst case error analysis is done by taking 
the worst possible q. 

By symmetry, (2) and (3) are equal assuming the distortion function is ad¬ 
ditive (more generally, permutation invariant) and this leads to a proof that 
C > Ru{D). The above steps will be discussed in much detail, later in this 
paper. This equality of (2) and (3) is a randomized covering-packing con¬ 
nection, and is a duality between source-coding and channel-coding. Further, 
this is an operational view and proof in the sense that only the operational 
meanings of channel capacity as the maximum rate of reliable communica¬ 
tion and the rate-distortion function as the minimum rate needed to com¬ 
press a source with certain distortion are used. Of course, certain randomized 
codebook constructions are used. No functional simplifications beyond the 
equality of (2) and (3) are needed. 

This proof is discussed precisely in Section 4 and intuitively in Appendix A. 

If b is the composition of an encoder, channel and decoder, that is, b^ = e”o 
kof^ for some encoder, decoder, < e”, /” >“ and channel k and the uniform 
X source is communicated over this channel by use of some encoder-decoder 

< FI", F" >“. Then, it follows that by use of encoder-decoder < F"oe", /"o 
F" >“, reliable communication can be accomplished over channel k at rates 

< Ru{D). By use of the argument of source-coding followed by channel¬ 
coding, optimality of source-channel separation for communication of the 
uniform X source over the channel k. This leads to an operational view of 
source-channel separation for communication with a fidelity criterion. Note 
that both the channel capacity problem and the rate-distortion problem 
are infinite dimensional optimization problems. By use of this methodology, 
the optimality of source-channel separation is proved without reducing the 
problems to finite dimensional problems. This is as opposed to the proof 
of separation, for example, in [2] which crucially relies on the the single¬ 
letter maximum mutual information expression for channel capacity and the 
single-letter minimum mutual information expression for the rate-distortion 
function. 

Since the decoding rule for the channel-coding problem depends only on the 
end-to-end description that the channel communicates the uniform X source 
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within distortion D, in addition to a general channel, assuming random 
codes are permitted, duality also holds for a compound channel, that is, 
where the channel belongs to a set, (see for example [3] for a discussion 
on compound channels). Note that the channel model is still general. For 
the same reason, source-channel separation for communication with a fidelity 
criterion also holds for a general, compound channel assuming random codes 
are permitted. This will be discussed in some detail, later. 

An operational view, as regards this paper, refers to a view which uses only 
the operational meanings of quantities: for example, of channel capacity as 
the maximum rate of reliable communication or the rate-distortion function 
as the minimum rate needed to code a souree with a certain distortion It 
does not mean constructive. 

The source U is ideal for this purpose because it puts mass only on the 
set of sequences with a particular type. If one tries to carry out the above 
argument for the i.i.d. X source, es and (5s enter the picture. A generalization 
to the i.i.d. X source can be made via a perturbation argument. 


2. Literature Survey 


Duality between source-coding and channel-coding has been discussed in a 
number of settings in the information-theory literature. 

Shannon [2] discussed, on a high level, a functional duality between source¬ 
coding and channel-coding by considering a channel-coding problem where 
there is a cost associated with different input letters which amounts to find¬ 
ing a source which is just right for the channel and desired cost. Similarly, 
the rate-distortion source-coding problem corresponds to finding a channel 
that is just right for the source and the allowed distortion level. Further, 
Shannon makes the statement, “This duality can be pursued further and 
is related to a duality between past and future and notions of control and 
knowledge. Thus we may have knowledge of the past but cannot control it; 
we may control the future but have no knowledge of it.” 

A general formulation of this functional duality has been posed in [4] which 
considers the channel capacity with cost constraints problem and the rate- 
distortion problem, defines when the problems are duals of each other, and 
proves that channel capacity is equal to the rate-distortion function if the 
problems are dual. The purpose of our paper is not a functional duality or a 
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mathematical programming based duality, but a operational duality where 
operational is defined in the previous section. 

Operational duality, as defined by Ankit et al [5] refers to the property that 
optimal encoding/decoding schemes for one problem lead to optimal encod¬ 
ing/decoding schemes for the corresponding dual problem. They show that 
if used as a lossy compressor, the maximum-likelihood channel decoder of 
a randomly chosen capacity-achieving codebook achieves the rate-distortion 
function almost surely . Note that the definition of operational used in [5] is 
different from the definition of operational used in this paper. 

Csiszar and Korner [3] prove the rate-distortion theorem by hrst constructing 
a “backward” DMC and codes for this DMC such that source-codes meeting 
the distortion criterion are obtained from this channel code by using the 
channel decoder as a source encoder and vice-versa; for this purpose, channel 
codes with large error probability are needed. The view-point is suggestive of 
a duality between source and channel coding. There is no backward channel 
in our paper: there is a forward channel which directly communicates the 
source U within distortion D and there is the rate-distortion source-coding 
problem. 

Yassaee [6] have studied duality between channel coding problem and secret- 
key agreement problem (in the source-model sense) They show how an 
achievability proof for each of these problems can be converted into an 
achievability proof for the other one. 

The decoding rule used in this paper is a variant of a minimum distance 
decoding rule. For discrete memoryless channels, decoders minimizing a dis¬ 
tortion measure have been studied as mis-match decoding and are subop- 
timal in general though optimal if the distortion measure is matched, that 
is, equal to the negative log of the channel transition probability; see for 
example the paper of Csiszar and Narayan [7]. 

The results in this paper form a part of the first authors Ph. D. dissertation 

[ 8 ]. 

Recall the important point that the duality between source-coding and 
channel-coding, as discussed in this paper is operational in the sense it uses 
only the operational meanings of channel capacity as the maximum rate of 
reliable communications and the rate-distortion function as the minimum 
rate needed to code a source with certain distortion levels, and this sense 
is different from the sense in which duality is discussed in the above men¬ 
tioned papers. Major functional simplifications are not used. Random codes 
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are constructed for both problems and a connection is seen between the two 
problems, which leads to a randomized covering-packing duality. 


3. Notation and definitions 


Superscript n will denote a quantity related to block-length n. For example, 
will be the channel input when the block-length is n. As block-length 
varies, x =< x” >“ will denote the sequence for various block-lengths. 

The source input space is X and the source reproduction space \sy.X and 
y are finite sets. X is a random variable on X. Let px{x) be rational Vx. Let 
no denote the least positive integer for which nopx{x) is an integer Vx G X. 
Let denote the set of sequences with (exact) type px- is non-empty 
if and only if no divides n. Let n' = non. Let C/” denote a random variable 
which is uniform on and zero elsewhere. Then, < [/"■ >“ is the uniform 
X source and is denoted by U. The uniform X source can be defined only 
for those X for which px{x) is rational Vx G X. 

Every mathematical entity which had a superscript n in Section 1 will have 
a superscript n' henceforth. This is because the uniform X source is defined 
only for block-lengths n'. The reader is urged not to get confused between this 
change of superscript between Section 1 and the rest of this paper. Further, 
the reader is urged to read Section 1 by replacing n with n' in mathematical 
entities. 

Let q denote a type on the set y which is achievable when the block-length 
is n'. Vg is the set of all sequences with type q. The uniform distribution 
on V"' is V^'. 

Since the uniform X source is defined only for block-lengths n', distortion 
function, channels, encoders and decoders will be defined only for block- 
lengths n'. 

d =< d^' >“ is the distortion function where dT' : X'^' x y^' —)• [0, oo). 
Let tt"' be a permutation (rearrangement) of (1,2,..., n'). That is, for 1 < 
i < n', (i) G {1, 2,..., n'} and that, 1 < i < n' are different. For 

g ^n' ^ denote 
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For y"’ G , tt” is defined analogously. < d"’ >“ is said to be permu¬ 
tation invariant if Vn', 


(5) 


(tt’" x^',7r^ y^) = d^ (x” ,y” ),Vx” € ,y^ G 


An additive distortion function is defined as follows. Let d : X x y ^ [0, oo) 
be a function. Define 


n' 

(6) d-'(x-',y"') = J]d(x"'(i),y-'(i)) 

i=l 

Then, < dP >f‘ is an additive distortion function. 

Additive distortion functions are special cases of permutation invariant dis¬ 
tortion function. Except at the end of the paper where conditions are derived 
for a certain technical conditions to be true for which additive distortion 
functions will be required, most of this paper will use permutation invariant 
distortion functions. 

A general channel h =<h^ >“ is defined as follows: 

The input space of the channel is X and the output space is y. 

(7) b^' :X^' ^V{y^’) 

xn' ^ 

jjTi ^yU i^n ^ should be thought of as the probability that the output of the 
channel is y” given that the input is x” . 

Note that the channel model is general in the sense of Verdu and Han [1]. 
Let 

(8) ^{1,2,...,2L”'«J} 

is the message set. When the block-length is n', a rate R determinis¬ 
tic source encoder is e^' : X"'' —)• Al^' and a rate R deterministic source 
decoder f^' : —)• y^'. is the block-length n' rate R deter¬ 

ministic source-code. The source-code is allowed to be random in the sense 
that encoder-decoder is a joint probability distribution on the space of de¬ 
terministic encoders and decoders. < e” , /” >“ is the rate R source-code. 
The classic argument used in [2] to prove the achievability part of the rate- 
distortion theorem uses a random source code. 
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When the block-length is n', a rate R deterministic channel encoder is a 
map e” : —)• X"' and a rate R deterministic channel decoder is a map 

: y^' —>■ where = A4^ U {e} is the message reproduction set 

where ‘e’ denotes error. The encoder and decoder are allowed to be random 
in the sense discussed previously. < e” , /” >“ is the rate R channel code. 
The classic argument used in [9] to derive the achievability of the mutual 
information expression for channel capacity uses a random channel code. 

The source-code < >“ is said to code the source U to within a 

distortion D if with input to e” o /” , the output is T” such that 

(9) lim Pr ([/”',y”') > = 0 

n’^oo J 

(9) is the probability of excess distortion criterion. The infimum of rates 
needed to code the uniform X source to within the distortion D is the 
rate-distortion function R^{D). If lim in (9) is replaced with lim inf, the 
criterion is called the inf probability of excess distortion criterion and the 
corresponding rate-distortion function is denoted by R^{D,inf). 

Denote 

( 10 ) g=<g-'>^^<e^^'ob^'of^'>r 

Then, g is a general channel with input space and output space ■ 
Rate R is said to be reliably achievable over h if there exists a rate R channel 
code < >“ such that 

(11) lim sup {{rrA ) = 0 

Supremum of all achievable rates is the capacity of b. 

The channel b is said to communicate the source U directly within distortion 
D if with input to b "^', the output is Y^' such that 

(12) lim Pr f ^d’"'(17’^',y’^') > D ) = 0 

n'^oo \^n ) 

See Figure 1 in Section 1 with n replaced by n'. 

In this paper, only the end-to-end description of a channel < 5"' ^ which 
communicates the uniform X source directly within distortion D is used 
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and not the particular 6” ; for this reason, the general channel should be 
thought of as a black-box which communicates the uniform X source within 
distortion D. 

In order to draw the randomized covering-packing duality between source 
and channel coding, the source-coding problem which will be considered 
is that of coding the source U within distortion D and the channel coding 
problem which will be considered is the rates of reliable communication over 
a b which communicates the source U directly within distortion D. A relation 
will be drawn between the rate-distortion function for the uniform X source 
and the capacity of b and in the process, the randomized covering-packing 
duality will emerge. 

4. Randomized covering-packing duality 

Theorem 1. Let b directly communicate souree U within distortion D un¬ 
der a permutation invariant distortion funetion d. Assume that R^{D) = 
Then, reliable communication can be accomplished over b at 
rates < R^{D). In other words, the capacity ofb, C > R^{D). 

Note that the technical condition R^{D) = R^{D,inf) can be proved for 
an additive distortion function. See the discussion following the proof of the 
theorem. 

Proof. This will be done by use of parallel random-coding arguments for 
two problems; 

• Channel-coding problem: Rates of reliable communication over b. 

• Souree-coding problem: Rates of coding for the uniform X source with 
a distortion D under the inf probability of excess distortion criterion. 

Codebook generation: 

• Codehook generation for the ehannel-coding problem: Let reliable com¬ 
munication be desired at rate R. Generate sequences indepen¬ 

dently and uniformly from . This is the codebook KT . 

• Codebook generation for the souree-coding problem: Let source-coding 

be desired at rate R. Generate codewords independently and 

uniformly from V” for some type q ony which is achievable for block- 
length n'. This is the codebook . 
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Joint typicality: 

Joint typicality for both the channel-coding and source-coding problems is 
defined as follows: {u^' ,y^') G x 3^"' jointly typical if 

(13) 

n 


Decoding and encoding: 

• Decoding for the channel-coding problem: Let be received. If there 

exists unique u” G /C"" for which {u^ , ) jointly typical, declare that 

is transmitted, else declare error. 

• Encoding for the source-coding problem: Let tt” G need to be 

source-coded. If there exists some y” G , encode to one such 

y” , else declare error. 

Some notation: 

• Notation for the channel-coding problem: Let message m"' G be 

transmitted. Codeword corresponding to m"^' is uf'. Non-transmitted 
codewords are u'f , u '2 , ■ ■ ■, is a realization of . Uff 

is uniform on . u'^ is a realization of U'^ . U'^ is uniform on , 
1 < i < 21-"''^-! — I. UJf', U'"' , 1 < i < 21-"''^-! — 1 are independent of 
each other. The channel output is y” . y^ is a realization of . y"" 
may depend on u^' but does not depend on u'^ , 1 < i < — 1. As 

random variables, Y'^ and Uff might be dependent but T” , , 1 < 

i < 2i"'— 1 are independent. If the type q of the sequence y”' needs 
to be explicitly denoted, the sequence is denoted by y” . Q'^ is the set 
of all achievable types g on T for block-length n'. 

• Notation for the source-coding problem: u'f is the sequence which needs 

to be source-coded, is a realization of C/” which is uniformly dis¬ 
tributed onU^'. The codewords are y”), 1 < i < where q denotes 

the type, y”) is a realization of I < i < 2 '^'^'^'^ where is uni¬ 
formly distributed on the subset of consisting of all sequences with 
type q. <',y”),l < i < are independently generated; as random 

variables, U^',Y^l, 1 <i < are independent. is the set of all 

achievable types g on T for block-length n' 
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Error analysis: For the channel-coding problem, the probability of correct 
decoding is analyzed and for the source-coding problem, the probability of 
error is analyzed. 


• Error analysis for the channel-coding problem: From the encoding¬ 
decoding rule, it follows that the event of correct decoding given that 
a particular message is transmitted is 


(14) 


,Y^)<D}nn 


2L"'«J-1 / 1 


'*=1 


-d” ([/'" ,Y^)>D 
n 


• Error analysis for the source-coding problem: From the encoding-decoding 
rule, it follows that the error event given that a particular message 
needs to be source-coded is 


(15) 


2L"'-RJ 

nil 


-d- (u^ 
n' 




Note that there is choice of q for codebook generation. 


Calculation: 


Calculation of the probability of correct decoding for the channel-coding 
problem: 

Bound for probability of event (14): 
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Pr Qld-'(c/-',y”') < D^nntT~" ,y^') > 

= Pr ,y"') < +Pr > nj) - 

Pr ,y’^') < u {U'^',Y^') > D 

>(1 -CUnO + Pr |^d"'(t/T>"') > ^}) - 1 

= - ^n' + Pr ,y-') > zi|) 

2L'n^Hj _2 

= -v„.+ n Pr((V(f/f.r”')>D|) 


(since U'^ , 1 < i < — 1, y^' are independent random variables) 


= -uJn'+ Pr N -d^iU^,Y^)>D 


2L"'hJ_i 


(where is uniform on and is independent of Y^ ] 
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— — ^n' + 


— — Wn' + 


— — + 


Py„>iy^')FT(^r'{U^\Y^')>D 
'ey-' \ 

Y1 PY-'iy"')P^{^d^'iu^',y"')>D 

'ey"' \ 

Py.'(y"')Pr(^id"'(C/-',y-')>Zl) 


=y^ 

yn' ^ yn' 
-. 2L"'«J-1 


-, 2L"'«J-1 


-, 2L"'«J-1 


Ly" ey- 

(since and y” are independent) 


> — Uln' + 


— — l^n' + 


inf Pr ( <j ,y^') > D 


Lr ey 


inf Vx[\-^d^'{U'-',y^‘)>D 
q&g-' ' I 


n' 


2 L-*'J?J_i 


2L"'hJ_i 


The last equality above follows because 


(17) 


PrQid’*'([/-',2/-') 


depends only on the type of y"^ ; see the symmetry argument later. 
Rate R is achievable if 

(18) 

2L"'hJ_i 

—)■ 1 as —>• oo 


— LOn' + 


inf 

geg"' 


Pr 


I n' 


yg 


)>D 


Since ujn' —)• 0 as n' —>• oo, rate R is achievable if 


(19) 


inf Pr 

qeG"' 


-d-{U\y-)>D 


-I 2 L"'«J -1 


1 as n' —)• oo 


Calculation of probability of error for the source-coding problem: 
Bound for probability of event (15) is calculated using standard argu- 
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merits: 

( 20 ) 


2L"'^J 


where is uniform on Vg . 

There is choice of G . Thus, a bound for the probability of the 
event is 


( 21 ) 


inf Pr 

geS"' 




2L"'hJ 


Since the inf probability of excess distortion criterion is used, it follows 
that rate R is achievable if 


( 22 ) 

inf Pr 

q&Q"'' 




n. 


' ^ q 


)>D 


0 for some n( = nou-i, 


m 


oo 


Connection/Duality between channel-coding and source-coding: 

The calculation required in the channel-coding problem is 

(23) ^mf ^ Pr ^d"'(17"', y^"') > 
and the calculation required in the source-coding problem is 

(24) inf Pr f I ^d^'{u^',Vf) > D 

q&g^’ \[n' 

It will be proved that (23) and (24) are equal. It will be proved more generally 
that 



n 


-d^'{U^',y^')>D 


= Pr 


n 


-d^'{u^',V^') > D 


(25) Pr 
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This is a symmetry argument and requires the assumption of permutation 
invariant distortion function. The idea is that the left hand side of (25) 
depends only on the type of y” . From this it follows that the left hand side 
of (25) is equal to 

(26) PrQid-'(C/-',<)>Zl|) 

where is independent of . Similarly, the right hand side of (25) de¬ 
pends only on the type of u” and from this it follows that the right hand 
side of (25) is also equal to (26). (25) follows. Details are as follows: 

First step is to prove that 

(27) Pr > d}) = Pr j,-') > d|) 

for sequences and with type q. Since C/" is the uniform distribution 
on 7/"', it follows that it is sufficient to prove that the sets 

(28) : ^r'{u^',y^') > d| and ,y'/) > d| 

have the same cardinality, y'^^ = vr"' y^ for some permutation tt” since y'^^ 
and y” have the same type. Denote the sets 

(29) 

Set B is defined analogously. 

Uq 

Let vP' G Byr ,'. Since the distortion function is permutation invariant, dP (vr"' vP , tt"’ y” ) 
= d^'{u^',yf). Thus, TT^'u^' G By,^„'. If u^' + u'^', It fol¬ 

lows that \By,r,' I > \By^>\. Interchanging y” and y^” in the above argument, 

\Byr,'\ > \B It follows that \B^ = \B (27) follows. 

y q y q y 9 y q 

Let Vq be independent of U'^'. From (27) it follows that 

({^d"'(C/"',y”') >^}) = PrQ^d"'([/"', >d|^ 


(30) Pr 
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By an argument identical with the one used to prove (27), it follows that 

(31) =Pr(|ld”'(«'"',V'”')>I)|) 
for u^' ,u'^' € W^'. From (31) it follows that 

(32) Pr Qld-'(«->-') > = Pr ld”'(7/-', y"') > Ilj) 

From (30) and (32), (25) follows. 

Proof that a channel which is capable of communicating the uniform X 
source with a certain distortion level is also capable of communicating bits 
reliably at any rate less than the infimum of the rates needed to code the 
uniform X source with the same distortion level under the inf probability of 
excess distortion criterion: 

Denote 

(33) 

An' = inf^Pr = inf,Pr {u^',V^') > d| 

From (19), it follows that rate R is achievable for the channel-coding problem 
if 

(34) > 1 as n' —)■ oo 

From (22), it follows that rate R is achievable for the source-coding problem 
if 

(35) [An'.)'^'^ ‘ ^ 0 as n( —)■ oo for some n( = non* for some n, —)• oo 

Let 

(36) a = sup{i? I (34) holds} 

Then, if R' > a, 

(37) lim {An'.)^'^ ' < 1 V i?' > a for some sequence n( — )> oo 

n'-^oo * 
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n' may depend on E!. 

Then, 

(38) lim = 0 for R" > R! 

n\—^oo * 

(37) and (38) hold for all i?" > R' > a. It follows that rates larger than a 
are achievable for the source-coding problem. 

Thus, a channel which is capable of communicating the uniform X source 
with a certain distortion level is also capable of communicating bits reliably 
at any rate less than the infimum of the rates needed to code the uniform 
X source with the same distortion level under the inf probability of excess 
distortion criterion. 

Wrapping up the proof of the theorem: 

It follows that if source U is directly communicated over b within distor¬ 
tion D, then reliable communication can be accomplished over b at rates 
< R^{D,inf). By use of the assumption R^{D) = R^{D,inf), it follows 
that reliable communication can be accomplished over b at rates < R^{D). 
In other words, the capacity of 6, C > R^{D). □ 

5. Discussion and recapitulation 

Randomized code constructions were made for a source-coding problem and 
a channel-coding problem and relation drawn between source-coding rates 
and channel coding-rates for the two problems. The source-coding problem 
is a covering problem and the channel-coding problem is a packing problem. 
For this reason, the connection is a randomized covering-packing connection. 
This duality between source-coding and channel coding is captured in (25). 

Note Berger’s lemma or the type covering lemma [3], that at least for ad¬ 
ditive distortion functions, there exist source codes of rates approaching 
R^{D) such that “balls” around codewords cover all sequences of type px, 
not only a large fraction of them. Thus, in (9), one does not need to take 
a limit; in other words, in the source-coding problem, one may not need 
to take a limit. Thus, a deterministic version of the source-coding prob¬ 
lem is possible; however it is unclear, how to do the same for the channel¬ 
coding problem. For this reason, the randomized versions of the problems 
are needed. 
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The technical condition R^{D) = is made on the rate-distortion 

function. This technical condition holds for additive distortion functions, and 
an operational proof which uses code constructions and various properties 
and relations between code constructions is provided in Chapter 5 of [8]. 

A proof of source-channel separation for communication with fidelity crite¬ 
rion follows as follows: If there exist encoder-decoder < e"', /” >“ such 
that by use of this encoder-decoder, communication of source U within 
distortion D happens over a channel k, then, b =< o k o >“ is 
a channel which communicates the source U directly within distortion D. 
Thus, rates < R^{D) are achievable over b by use of some encoder-decoder 
< ,F^ >“. For this reason, reliable communication is possible over k 

at rates < R^{D) by use of encoder-decoder < o e^', f^' o E"^' >“. By 
use of the standard argument of source-coding followed by channel coding, 
if capacity of /c is > R^D), the uniform X Source can be communicated 
over k by source coding followed by channel coding. Proof of separation fol¬ 
lows. The proof only uses the operational meanings of capacity (maximum 
rate of reliable communication) and rate-distortion function (minimum rate 
needed to compress a source with certain distortion), and randomized code 
constructions for these problems instead of using finite-dimensional func¬ 
tional simplifications or finite dimensional information theoretic definitions, 
for example, capacity as maximum mutual information and rate-distortion 
function as minimum mutual information, unlike in the traditional proof of 
Shannon [2]. Functional simplifications are carried out to the extent of (25). 

Note that whether a view or a proof is operational (in the sense used in this 
paper) cannot be defined mathematically precisely. However, the same can 
be sensed intuitively from the context in which it is used. 

By use of a perturbation argument, the results can be generalized to the i.i.d. 
X source (general px^ not necessarily those for which px{x) is rational) for 
additive distortion functions as discussed in Chapter 5 of [8]. 

Finally, note that the argument to prove Theorem 1 uses random codes. 
However, if the channel is a single channel, existence of a random code im¬ 
plies the existence of a deterministic code. Note further, that in the decoding 
rule in Theorem 1, only the end-to-end description that the channel com¬ 
municates the uniform X source within distortion D is used, and not the 
particular < b'^ ^ . For this reason, even if the channel belongs to a set, 
that is, the channel is compound in the sense of [3], Theorem 1 still holds. 
However, random codes would be needed since the argument to go from a 
random code to a deterministic code does not hold for a compound channel. 
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For the same reason, a universal source channel separation theorem for com¬ 
munication with a fidelity criterion where universality is over the channel 
(channel is compound) holds if random codes are permitted. Precise details 
of a general, compound channel, what it means for a general, compound 
channel to communicate the uniform X source within distortion D, and the 
capacity of a general, compound channel, are omitted. 
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Appendix A. Intuitive explanation of the randomized 

covering-packing duality 


This appendix explains on an intuitive level, the covering-packing duality. 
The authors emphasize that mathematically this section is imprecise and is 
only for the purpose of developing intuition. 

A general channel which directly communicates the uniform X source to 
within a distortion D can be intuitively thought of as follows: with high 
probability, a sequence in is communicated with distortion < nD' and 
this probability —)• 0 as n' —)• oo. See Figure 2 in Section 1 with n replaced 
with n'. 

The deterministic (as opposed to randomized) covering-packing, or the source 
coding-channel coding problem in our setting, on an intuitive level is pic¬ 
tured in Figure 3. For the covering problem, with reference to this figure, 
yU g yn balls of radius n'D around are made in the space. 
The source-coding question is: what is the minimum number of balls of ra¬ 
dius n'D which cover the space. In other words, what is the minimum 
number of G T” such that balls of radius nD around these y” G T”' 
cover the 14^' space. For the packing problem, first recall the intuitive action 
of the channel depicted in Figure 2. With reference to the figure, for the 
packing problem, the question is to find the minimum number of what is 
the maximum number of tt” G such that balls of radius n'D around 
these pack the T"" space. 

The randomized covering-packing picture is figuratively described in Figure 
4. 

In the covering problem, let the block-length be re'. Suppose re”' needs to be 
compressed. Suppose a codeword of type precisely q is generated uniformly 
from the set of all sequences with type precisely q. Denote this uniform 
distribution by and a realization of by y” . Probability that y” will 
code [/” is 


(39) 



This probability is independent of re” by symmetry because the distortion 
metric is 


(40) 
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Covering: 



• is a point G 0 Codeword in y^' 


Packing: 



Figure 3: Covering: what is the minimum number of balls (equivalently, 
the number of codewords G y^') with centers around certain y^' G y^' and 
balls in which cover the whole space. Packing: what is the maximum 
number of balls with (equivalently, the number of codewords G U'^') centers 
around certain u” G U^' such that these balls pack the space. Note that 
balls in the covering problem have centers G y'^ but the balls are in 
whereas balls in the packing problem have centers G but the balls are 
in y^' 
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The way things intuitively work for increasing block-lengths, the number of 
sequences needed to code the source C/" if codewords of type q are used is 
approximately 


(41) 


1 

Pr ([/«', F-') <^) 


q is arbitrary and thus, with this coding scheme, the number of codewords 
to code the uniform X source is approximately 

(42) inf - ;r-j -——;---^ = j3 

<1 <D) 

In general, there may be a scheme for which number of codewords needed 
is < 13. 

In the packing problem, generate 2"''^ codewords independently and uni¬ 
formly from . Suppose is transmitted. By the action of the channel, 
it follows that with high probability, y” is received such that 

(43) 

n 

Let the type of the received sequence y"" be ( 7 . Let be another non- 
transmitted codeword which is generated using U’'^ . Note that and 
U'^ are the same in distribution. Probability that there might be a mistake 
to say that u'” is transmitted is 

(44) ,y^')<D 

n 

The above probability is the same for all y” by symmetry because the 
distortion metric is permutation invariant, and hence, is equal to 

(45) Pr ,V^') < 

where is defined in the above discussion on covering. Note that q is arbi¬ 
trary and in order to get a bound on the total number of allowed codewords, 
the worst possible q needs to be considered. The way union bound works 
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and the way things work for large block-lengths, the number of sequences 
which can be chosen as codewords for the channel-coding problem is 


(46) 


inf 

<? 


1 

Pv {^d^'{U'^',Vg^') < D) 


13 


In general, there may be a scheme for which number of codewords is > [3. 

Finally, note that the /3 in the covering and packing problem are the same. 
It follows that C > R^{D) where C is the capacity of the channel. 

This is the intuitive base behind the proof of Theorem 1 and the resulting 
duality. Note further that this section is only for the sake of intuition and is 
mathematically imprecise. Precise proof have been provided in the proof of 
Theorem 1. 
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Randomized covering: 



Randomized packing: 



Figure 4: The randomized covering-packing picture for the problem of com¬ 
munication with a fidelity criterion 













